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Preliminary  Findings:  Design  of  Experiments 

Dr.  J.  Wesley  Barnes,  MAJ  J.  Whilden,  and  J.  Toufic  Chahin 
Institute  for  Advanced  Technology,  The  University  of  Texas  at  Austin 


University  XXI  Background 

The  Institute  for  Advanced  Technology  (lAT),  an  Army  University  Affiliated  Research 
Center  (UARC)  operating  under  US  Army  contract,  was  founded  in  1990  as  an  autonomous 
research  unit  of  The  University  of  Texas  at  Austin  (UT)  under  the  Office  of  the  Vice  President 
for  Research.  The  I  AT  has  been  in  continuous  operation  since  then,  first  as  a  Federally  Funded 
Research  and  Development  Center,  and  then  as  an  Army  UARC  from  1993  to  the  present, 
conducting  basic  research  in  the  areas  of  hypervelocity  physics,  electrodynamics,  pulsed  power, 
and  technical  education.  The  lAT  research  efforts  include  applied  research  through  the 
University  XXI  (UXXI)  program,  which  assists  the  military  to  address  issues  relating  to 
digitization  and  military  transformation. 

As  a  part  of  the  lAT,  UXXI  has  access  to  world-class  professors  and  graduate  students  at  UT 
in  order  to  conduct  its  projects.  The  faculty  and  staff  comprise  proven  leaders  whose  knowledge 
covers  a  broad  range  of  disciplines,  from  academic  and  industrial  research  to  test,  evaluation,  and 
production.  The  lAT  also  offers  a  Secret  facility  clearance,  with  classified  phone  and  fax,  and 
extensive  classified  and  unclassified  computing  facilities.  This  grants  UXXI  an  ability  to  work 
on  a  number  of  projects  that  would  otherwise  be  off  limits  to  typical  university  level  research 
facilities,  including  the  work  done  on  the  Design  of  Experiments  project. 


Design  of  Experiments  Project  Background 

As  the  United  States  faces  accelerating  military  challenges,  the  US  Army  must  make  the  best 
use  of  available  resources.  Narrow  windows  of  time,  limited  troop  availability,  and  limited 
funding  present  a  special  challenge  for  the  US  Army  Operational  Test  Command  (USAOTC). 
Existing  test  operating  procedures  and  methodology  (TOPM)  do  not  incorporate  design  of 
experiment  (DOE)  techniques.  These  techniques  are  currently  utilized  by  the  US  Air  Eorce 
Operational  Test  &  Evaluation  Center  (AEOTEC),  resulting  in  better  time  utilization  and  cost 
savings.  USAOTC  tests  with  up  to  multiple  battalions  are  substantially  more  complex  than 
AEOTEC  tests  with  pairs  of  planes,  so  a  simple  transfer  of  techniques  will  not  completely 
address  USAOTC  needs.  A  solution  tailored  to  US  Army  scenarios  is  needed. 

The  Design  of  Experiments  project  is  developing  recommendations  for  a  new  TOPM  for 
applying  DOE  Techniques  to  the  US  Army’s  complex  operational  testing  environment.  These 
recommendations  are  being  based  upon  evaluations  of  existing  event  design  plans  and  testing 
reports.  The  focus  has  been  primarily  upon  small-  to  medium-scale  testing  scenarios  in  order  to 
provide  recommendations  that  are  easy  to  understand.  The  inherent  complexity  of  large-scale 
testing  events  also  increases  the  difficulty  of  making  clear  recommendations  for  improvements  in 
testing.  It  is  hoped  that  through  the  development  of  case  studies  and  leveraging  AEOTEC’s 
work,  we  will  develop  recommendations  that  tailor  DOE  techniques  and  exploit  their  benefits  for 
use  by  operation  research  analysts  with  USAOTC  scenarios.  These  statistical  techniques  are 
expected  to  afford  opportunities  for  substantial  savings  in  resources  for  conduct  of  operational 
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testing  as  well  as  aeeommodate  testing  of  eomplex  large  systems.  As  we  look  to  the  testing  of 
future  eomplex  systems  of  systems  (SoS),  these  reeommendations  will  offer  a  solution  for  low- 
eost,  rapid-turnaround  integrated  testing. 


Assessment  of  Current  Practices 

This  doeument  will  serve  as  an  interim  summary  of  researeh  to  date  on  how  the  Army 
eonduets  testing  on  potential  new  or  modified  equipment  for  fielding  to  soldiers.  This  summary 
will  not  make  any  reeommendations  but  will  show  what  areas  will  be  foeused  on  for  a  researeh 
paper  of  mueh  larger  seope,  to  be  delivered  in  April  2007.  MAJ.  John  Whilden  has  been  working 
direetly  with  the  Close  Combat  (CC)  and  the  Aviation  (AV)  test  direetorates  at  Operational  Test 
Command  (OTC).  Both  the  CC  and  AV  direetorates  have  provided  him  with  additional 
doeuments  for  review.  The  Methodology  and  Analysis  Division  (MAD)  at  OTC  eontinues  to 
provide  assistanee  as  needed.  The  researeh  has  been  expanded  to  inelude  the  evaluators  at  Army 
Test  and  Evaluation  Command  (ATEC). 

There  is  a  some  resistanee  to  ehange  at  OTC,  an  attitude  found  in  many  large  institutions. 
Most  of  the  eivilian  ORSAs  and  aequisition  testers  have  predetermined  methods  of  evaluating 
projeets  set  before  them.  Many  of  the  eivilians  know  about  DOE  teehniques  but  have  yet  to 
embraee  them.  There  is  a  large  push  to  embraee  Eean  Six  Sigma  (ESS)  by  the  Army,  ineluding 
the  test  eommunity.  Part  of  ESS  is  proeess  reduetion.  While  MAJ.  Whilden  has  reeeived 
eomplete  support  at  the  upper  personnel  levels,  there  is  some  eoneem  at  the  lower  personnel 
levels. 

The  biggest  issue  voieed  by  testers  at  OTC  is  failure  to  follow  the  proper  testing  sequenee  as 
outlined  by  Army  Regulation  (AR)  73-1  and  ATEC  Pamphlet  73-1.  It  is  routine  that  operational 
testing  starts  before  an  approved  system  evaluation  plan  (SEP)  is  in  plaee,  despite  the  SEP’s 
being  required  by  regulation.  The  SEP  identifies  the  system  being  evaluated,  the  eritieal  issues, 
and  the  system’s  proposed  use  in  the  hands  of  soldiers.  It  is  this  doeument  that  OTC  uses  to  help 
formulate  event  design  plan  (EDP).  When  the  SEP  ehanges,  it  either  delays  testing  or  it 
introduees  error  into  OTC  testing,  and  it  is  not  uneommon  for  testing  to  be  underway  when 
ehanges  are  made. 

There  is  an  unelear  methodology  to  determine  the  appropriate  experiment  size  and  number  of 
iterations.  A  eommon  number  of  repetitions  at  OTC  seems  to  be  30  runs  using  a  dedieated 
eompany-sized  element  of  troops.  There  does  not  seem  to  be  any  statistieal  basis  for  determining 
the  size  or  number  of  repetitions  for  an  experiment.  The  only  software  that  the  test  direetorates 
have  available  are  Mierosoft  Exeel  and  Mierosoft  Aeeess.  Although  Exeel  does  have  statistieal 
add-ins  eapable  of  ereating  summary  statisties,  it  does  not  appear  to  have  the  eapability  to 
statistieally  determine  an  appropriate  experiment  size  and  proper  number  of  iterations.  DOE 
allows  eontrol  over  this  by  designing  the  experiment  size  based  on  an  aeeeptable  level  of 
eonfidenee  and  error. 

It  is  elear  that  OTC  does  not  use  any  method  that  allows  for  the  interaetion  of  variables. 
Whereas  identifying  interaetion  is  a  pronouneed  strength  of  DOE,  most  other  methods — 
ineluding  the  Taguehi  method  and  one  faetor  at  a  time  (OEAT) — eannot  ealeulate  variable 
interaetions.  Variable  interaetions  are  statistieally  attributed  as  a  main  effeet  of  a  single  variable 
that  ean  introduee  errors  into  the  test.  Stronger  variable  interaetions  lead  to  greater  error  using 
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OFAT  or  the  Taguchi  methods.  OTC  primarily  uses  Taguehi  and  OF  AT  type  methods  in  its 
testing  methods  and  designs. 

A  fundamental  eoneept  of  DOE  is  determining  which  variables  impact  the  process  and 
designing  an  experiment  that  looks  at  the  critical  variables.  Currently,  OTC  designs  experiments 
without  considering  the  impact  of  any  variables.  Tests  center  around  critical  performance  issues 
and  not  the  variables  that  impact  the  performance  of  the  equipment.  Once  again,  error  is 
introduced  in  experiments  when  critical  variables  are  ignored.  If  a  system  tested  does  poorly,  it  is 
difficult  under  current  test  design  to  determine  why;  likewise,  if  a  system  being  tested  performs 
well,  the  reasons  remain  unknown. 

One  difficult  aspect  for  OTC  is  the  fact  that  OTC  is  primarily  concerned  with  the  equipment 
in  the  hands  of  soldiers.  Developmental  Test  Command  (DTC)  looks  at  actual  laboratory-type 
data,  such  as  muzzle  velocity,  and  a  shot  group  under  ideal  conditions  that  omit  human  error.  A 
weapon  could  be  fired  by  a  computer  on  a  mechanical  rest.  DTC  conducts  benchmark  testing  to 
ensure  that  the  equipment  performs  to  specification.  DTC  weighs  ammunition  fired  and  has 
accelerated-aging  labs  for  testing  durability  to  give  two  good  examples.  Whereas  most  of  the 
data  collected  by  DTC  can  be  automated  with  extreme  accuracy,  most  of  OTC  testing  is  much 
more  subjective,  relying  on  the  input  of  soldiers. 

The  evaluation  team  at  ATEC  compiles  all  collected  data  from  DTC  and  OTC  and  then  runs 
statistical  processes  on  the  data  to  create  findings.  The  responsibility  of  OTC  is  to  provide 
packaged  data  that  are  accurate  and  in  the  form  of  spreadsheets  and  databases.  OTC  is  required 
at  times  to  provide  a  test  report  with  summary  statistics  including  graphs,  means,  quartiles, 
probable  errors,  etc.,  using  Excel.  The  evaluation  team  at  ATEC  uses  more  powerful  tools,  such 
as  SAS  Jump,  for  statistical  analysis.  One  further  area  of  research  will  look  at  any  problems 
created  by  not  having  the  testers  at  OTC  involved  in  the  evaluation  process  to  analyze  data. 


An  OTC-Provided  Scenario  Amended  to  Use  Design  of  Experiments 

You  are  participating  in  a  meeting  of  the  ATEC  System  Team  (AST)  for  the  Eong-Range 
Hunter  reconnaissance  system.  This  team  has  met  before,  and  its  members  have  provided  their 
input,  taking  into  account  information  from  the  test  and  evaluation  working  integrated  process 
team  (T&E  WIPT).  The  SEP  is  complete,  and  it  is  time  to  provide  the  test  designs  for  the  various 
test  events  to  detailed  planning.  One  of  these  events  is  the  nighttime  maneuver  portion  of  the 
initial  operational  test  (lOT),  which  is  the  primary  source  for  effectiveness  data  on  target 
detection  capabilities.  This  portion  of  the  lOT  is  an  instrumented  field  exercise  performed  under 
realistic  conditions  with  typical  user  troops.  Each  field  trial  is  four  hours  long,  under  specified 
conditions,  and  one  field  trial  per  night  is  planned  using  four  Eong-Range  Hunters.  The  troop 
unit  is  available  for  30  days.  The  Test  Officer  has  prepared  a  draft  test  design  for  this  event  and 
furnished  it  to  all  AST  members.  This  meeting  is  to  consider  any  final  changes  that  may  be 
required  before  including  this  test  design  in  the  Test  Plan  for  the  lOT.  The  Test  Officer  is  the 
AST  Chair  at  this  time,  and  will  open  the  meeting. 

Your  responsibility  is  to  make  sure  the  test  generates  the  needed  data  as  planned  in  the  data 
source  matrix  (DSM),  is  a  sound  and  realistic  tactical  exercise,  and  is  efficient  and  economical. 
You  are  the  chair  of  this  ATEC  System  Team,  and  you  prepared  this  draft  test  design  for  review 
by  the  team.  You  feel  that  three  field  trials  are  the  minimum  needed  in  each  set  of  conditions  to 
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generate  adequate  data.  You  have  planned  four  subtests,  eaeh  with  an  appropriate  number  of  sets 
of  three  trials.  Your  design  provides  for  at  least  three  data  trials  for  eaeh  test  eondition — for 
example,  three  trials  with  eountermeasures  and  three  without.  And  of  eourse,  there  should  be 
some  extra  trials  for  eontingeneies,  such  as  having  to  re-run  any  trials  that  are  not  valid. 

As  far  as  you  know,  this  design  is  acceptable  to  everyone  on  the  AST,  but  you  are  open  to 
suggestions  for  improvement.  If  anyone  in  the  team  sees  any  errors  or  opportunities  for 
improvements,  you  will  lead  the  effort  to  reach  consensus.  As  chair,  you  will  open  this  meeting 
with  a  call  for  comments  on  the  draft  design.  This  example  also  places  each  of  the  course 
attendees  in  a  different  role  with  different  information.  Appendix  4A  has  a  list  of  all  data, 
including  role-play  information. 


The  relevant  section  of  the  Long-Range  Hunter  SEP  is  Paragraph  4. 7. 7. 3: 


Subtest  1  -  Types  of  Tactical  Maneuver 

Conditions 

Offense 

Defense 

Retrograde 

Reinforcement 

Total 

Nr  of  Trials 

3 

3 

3 

3 

12 

Subtest  2  -  Means  of  Long-Range  Hunter  Operation 

Conditions 

Operated  from 
Vehicle 

Operated  by  Dismounted 
Personnel 

Total 

Nr  of  Trials 

3 

3 

6 

Subtest  3  -  Range  to  Presented  Targets 

Conditions 

Long  Range 

Short  Range 

Total 

Nr  of  Trials 

3 

3 

6 

Subtest  4  -  Countermeasures 

Conditions 

CM  used  by  OPFOR 

No  CM  used 

Total 

Nr  of  Trials 

3 

3 

6 

This  test  design  needs  30  data  record  trials  and  includes  six  contingency  trials. 
A  possible  OTC  course  solution  to  the  problem  is: 


Countermeasures  Present 

No  Countermeasures 

Off 

Def 

Retro 

Off 

Def 

Retro 

Veh 

3 

3 

3 

3 

3 

3 

18 

Dismt 

3 

3 

3 

3 

3 

3 

18 

6 

6 

6 

6 

6 

6 

18 

18 

36 

The  OTC  solution  suggests  that  the  improved  design  increases  the  number  of  trials  (sample 
size)  for  each  of  the  two  countermeasure  conditions  from  three  to  18,  increases  the  sample  size 
for  each  of  the  two  mounting  conditions  from  three  to  18,  and  increases  the  sample  size  for  each 
of  the  three  maneuver  conditions  from  three  to  12,  for  a  total  of  36  trials  overall.  (Range  is  not 
shown  in  the  matrix  because  it  is  a  tactically  varied  factor.)  Contingency  trials  are  not  needed 
because  each  condition  has  either  12  or  18  trials  and  could  lose  several  of  them  and  still  have  far 
more  than  the  required  number  of  trials. 
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Comments  on  the  OTC  solution  to  the  problem; 

•  There  has  to  be  a  statistieal  analysis  of  this  problem.  The  test  as  written  does  not  use 
statistieal  means  to  determine  the  correet  number  of  iterations  required  to  obtain  different 
acceptable  levels  of  a  and  P  error  with  a  calculated  level  of  confidence. 

•  This  solution  does  not  show  the  process  flow  of  the  Long-Range  Hunter  problem  to 
achieve  test  results  that  can  accurately  conclude  the  effectiveness  of  the  Long-Range 
Hunter  system.  The  process  flow  diagram  helps  to  show  what  is  actually  critical  to  the 
success  of  the  Long-Range  Hunter. 

•  The  most  critical  variable  (range)  is  left  off  the  solution.  It  is  perhaps  the  best  single 
variable  to  determine  the  actual  effectiveness  of  the  Long-Range  Hunter.  The  purpose  of 
the  Long-Range  Hunter  is  to  provide  a  picture  of  enemy  layout  on  a  battlefield.  A  Long- 
Range  Hunter  must  detect  the  enemy  before  being  detected  by  the  enemy. 

•  The  solution  indicates  the  use  of  one  factor  at  a  time  (OF AT)  in  the  test  experiment.  DOE 
allows  testing  to  look  at  the  interactions  of  variables.  A  key  interaction  that  would  be 
present  would  be  the  interaction  of  range  variables  and  countermeasure  variables.  The 
interaction  would  show  the  effectiveness  of  enemy  countermeasures  to  prevent  detection 
by  friendly  forces. 

More  detail  in  applying  DOE  methods  to  the  Eong-Range  Hunter  example  problem  will 
follow.  There  will  first  be  a  formulation  of  the  problem  using  the  four-step  DOE  method. 
Statistical  methods  will  be  used  to  determine  the  appropriate  number  of  iterations  per  variable. 
DOE  will  also  help  to  determine  the  appropriate  variables  to  test.  Once  the  test  is  designed,  a 
fictitious  database  of  test  results  will  be  analyzed  using  the  2*^  model.  Problem  formulation  using 
DOE  methods  will  use  the  same  factors  and  constraints  as  the  TEBC  course  solution  considered. 

The  process  flow  diagram  will  take  the  Eong-Range  Hunter  and  sculpt  an  experimental 
design.  The  process  flow  diagram  should  help  to  generate  a  process  flow  of  how  the  experiment 
will  test  the  Long-Range  Hunter,  and  it  should  generate  a  potential  list  of  measures  of 
performance  (MOPs)  that  testers  can  screen  to  determine  the  most  critical  variables. 

Eigure  1  is  a  sample  of  what  a  process  flow  diagram  (PED)  could  look  like.  Depending  on  the 
actual  mission  of  the  Eong-Range  Hunter,  you  could  develop  a  number  of  different  PEDs.  This 
PED  indicates  that  the  most  critical  aspect  of  the  Eong-Range  Hunter  is  its  ability  to  detect  the 
enemy  before  being  detected.  This  critical  concept  would  drive  the  development  of  MOPs  that 
would  be  critical  to  determine  if  a  system  meets  required  mission  capabilities  and  specifications. 
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Figure  1.  Sample  process  flow  diagram  (PFD). 

In  this  PFD,  the  Long-Range  Hunter  starts  at  a  predetermined  point  and  proeeeds  down  a 
speeified  maneuver  lane.  In  this  test,  enemy  targets  are  either  stationary  in  a  defensive  posture  or 
maneuvering  in  an  offensive  or  sereening  posture.  The  key  for  the  Long-Range  Hunter,  as  the 
yes/no  questions  show,  is  its  ability  to  deteet  the  enemy  before  being  deteeted.  The  PFD  allows 
for  several  different  situations,  ineluding  the  following; 

1 .  The  Long-Range  Hunter  deteets  the  enemy  before  being  deteeted. 

2.  The  Long-Range  Hunter  and  enemy  deteet  eaeh  other  near  simultaneously. 

3.  The  enemy  deteets  the  Long-Range  Hunter  before  it  deteets  the  enemy. 

4.  Neither  the  enemy  nor  Long-Range  Hunter  deteet  eaeh  other. 
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Each  scenario  represents  different  problems  that  the  test  should  diseern.  The  MOPs  help  to 
reeognize  the  overall  effeetiveness  of  a  system.  This  PFD  also  shows  that  the  Long-Range 
Hunter  ean  pass  parts  of  the  test  and  fail  others.  Failure  of  one  event  on  a  test  does  not  terminate 
the  test. 

The  following  table  is  a  list  of  sample  MOPs  that  testing  eould  use  to  determine  the  overall 
effectiveness  of  the  Long-Range  Hunter  as  a  system.  These  potential  MOPs  all  look  at 
performanee  aspeets  of  the  Long-Range  Hunter.  They  do  not  look  at  other  factors,  such  as 
maintenance  and  supportability  of  the  Long-Range  Hunter,  whieh  would  also  be  very  important 
(although  seeondary  to  aetual  mission  performanee). 


N 

MOP 

Units 

Range 

Priori  tv 

Data  Elem 

Sonrce 

1 

Detection  Distance  of 
Enemy  by  LRH 

M 

0-5000 

H 

Range  Instr  / 
OCs 

2 

Detection  Distance  of 
LRH  by  Enemy 

M 

0-5000 

H 

Range  Instr  / 
OCs 

3 

Maximum  Scan  Distance 
of  LRH 

M 

0-5000 

H 

OCs / LRH 
Crew 

4 

Time  to  engage  Enemy 
once  Detected 

S 

0-180 

M 

Radio  /  OCs 

5 

Target  Damage 

% 

0-100 

M 

MILES 

Report 

6 

Survivability  against 

Enemy  Fire 

% 

0-100 

M 

MILES 

Report 

7 

Time  of  LRH  to  break 
Contact 

S 

0-? 

M 

Radio  /  OCs 

8 

Detection  Angle  of  TGT 
from  LRH 

o 

0-360 

M 

OCs 

Units:  M  =  Meters  S  =  Seconds  %  =  Percentage  °  =  Degrees 
Priority:  H  =  High  M  =  Medium  L  =  Low 


Again,  understand  that  there  eould  be  any  number  of  MOPs,  depending  on  the  requirements 
of  the  tested  system.  These  MOPs  ean  be  prioritized  in  any  way  deemed  appropriate.  However,  if 
everything  is  high  priority,  then  nothing  is.  Data  for  these  MOPs  eould  eome  from  any  number  of 
available  sourees.  Obviously  the  more  automated  a  eolleetion  method  is,  the  less  human  error 
ean  ereep  into  the  test  results. 

Variable  construetion  beeomes  eritieal,  as  it  drives  the  overall  design  of  the  LRH  experiment. 
DOE  would  lay  out  variable  eonstruetion  as  shown  in  Figure  2. 
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Materials 


Measurements 

Manpower 

j 


While  Figure  2  presents  possible  variables,  a  refined  list  of  tested  variables  could  look  as 
follows: 


Tested  Variables 

1 

TGT  Presentation 

2000M 

500M 

2 

Mission  Type 

Offense 

Defense 

3 

Operation  Mode 

Mounted 

Dismounted 

4 

Enemy  Countermeasures 

Yes 

No 

5 

Enemy  Force 

Light  (INF) 

HVY  (Mech) 

These  variables  are  the  basis  for  DOE  construction.  The  OTC  construction  of  variables  does 
not  give  the  testers  any  way  to  know  which  variables  cause  the  success  or  failure  of  the 
experiment.  This  construction  of  variables  does.  Another  issue  with  the  OTC  solution  is  the  lack 
of  detail.  There  is  a  big  difference  between  presenting  targets  at  300  m  and  1000  m.  The  OTC 
model  does  not  capture  this.  The  OTC  model  ignores  weather,  terrain,  time  of  day,  illumination, 
etc.  These  variables  are  critical  to  determining  the  success  of  the  Long-Range  Flunter.  Under 
DOE,  choose  to  control  them.  Use  the  most  difficult  but  common  conditions  soldiers  could 
face — such  as  night,  offense,  open  and  wooded  terrain,  etc.  DOE  would  expect  control  of  these 
types  of  variables.  A  test  run  at  night  and  a  test  run  during  daylight  would  have  completely 
different  results.  A  test  run  with  50%  illumination  at  night  will  yield  different  results  than  a  test 
run  at  night  with  5%  illumination.  Eailure  to  control  these  types  of  variables  will  introduce  error 
into  an  experiment.  Using  these  five  tested  variables  yields  the  following  table  of  unique  runs: 
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Case 

TGT 

Pres 

Mission 

Type 

OP 

Mode 

Enemy 

CMs 

Enemy 

Force 

1 

Long 

Offense 

Mount 

Yes 

Light 

2 

Short 

Offense 

Mount 

Yes 

Light 

3 

Long 

Defense 

Mount 

Yes 

Light 

4 

Short 

Defense 

Mount 

Yes 

Light 

5 

Long 

Offense 

Dismount 

Yes 

Light 

6 

Short 

Offense 

Dismount 

Yes 

Light 

7 

Long 

Defense 

Dismount 

Yes 

Light 

8 

Short 

Defense 

Dismount 

Yes 

Light 

9 

Long 

Offense 

Mount 

No 

Light 

10 

Short 

Offense 

Mount 

No 

Light 

11 

Long 

Defense 

Mount 

No 

Light 

12 

Short 

Defense 

Mount 

No 

Light 

13 

Long 

Offense 

Dismount 

No 

Light 

14 

Short 

Offense 

Dismount 

No 

Light 

15 

Long 

Defense 

Dismount 

No 

Light 

16 

Short 

Defense 

Dismount 

No 

Light 

17 

Long 

Offense 

Mount 

Yes 

Heavy 

18 

Short 

Offense 

Mount 

Yes 

Heavy 

19 

Long 

Defense 

Mount 

Yes 

Heavy 

20 

Short 

Defense 

Mount 

Yes 

Heavy 

21 

Long 

Offense 

Dismount 

Yes 

Heavy 

22 

Short 

Offense 

Dismount 

Yes 

Heavy 

23 

Long 

Defense 

Dismount 

Yes 

Heavy 

24 

Short 

Defense 

Dismount 

Yes 

Heavy 

25 

Long 

Offense 

Mount 

No 

Heavy 

26 

Short 

Offense 

Mount 

No 

Heavy 

27 

Long 

Defense 

Mount 

No 

Heavy 

28 

Short 

Defense 

Mount 

No 

Heavy 

29 

Long 

Offense 

Dismount 

No 

Heavy 

30 

Short 

Offense 

Dismount 

No 

Heavy 

31 

Long 

Defense 

Dismount 

No 

Heavy 

32 

Short 

Defense 

Dismount 

No 

Heavy 

There  are  32  unique  runs  considering  these  five  two-level  variables.  These  32  runs  yield 
much  more  complete  data  than  the  suggested  36  runs  by  the  OTC  example.  DOE  also  offers  the 
capability  to  construct  a  partial  fractural.  If  the  enemy  force  variable  is  blocked  and  essentially 
controlled,  the  number  of  unique  runs  can  be  reduced  to  16  from  32.  If  enemy  countermeasures 
are  further  blocked,  the  number  of  unique  runs  can  be  reduced  to  8.  Blocking  might  assume  that 
a  heavy  enemy  force  was  used  and  that  there  would  always  be  enemy  countermeasures,  which 
would  be  the  worst  case.  A  partial  fractural  DOE  design  would  allow  you  to  reduce  the  number 
of  field  days  from  30  to  15.  Statistics  and  resources  would  drive  which  DOE  model  was  used. 
Blocking  the  variables  would  confound  them  and  could  introduce  error. 


Projected  Focus  of  In-Depth  Report 

It  is  projected  that  MAJ.  Whilden’s  overall  master’s  thesis  will  be  60  pages  in  length.  This 
will  cover  all  informational  topics  such  as  the  structure  of  ATEC,  applicable  regulations,  and  all 
topics  in  this  summary  report.  The  final  submission  is  still  set  for  the  last  week  of  April  2007. 
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This  report  will  hinge  upon  a  fictional  example  called  the  Long-Range  Hunter.  This  fictional 
example  is  used  by  OTC  to  train  newly  assigned  personnel  on  basic  test  design  methods.  The 
next  section  will  briefiy  demonstrate  the  efficiencies  gained  by  using  DOE. 

Future  research  will  focus  on: 

1 .  Showing  how  the  Army  can  implement  DOE  principles  to  reduce  error  in 
experiments,  thereby  reducing  required  assets. 

2.  Researching  the  impacts  of  testing  sequence  when  timing  problems  occur. 

3.  Showing  the  impacts  of  not  using  a  DOE  method  that  allows  interactions  of  variables. 

4.  Showing  how  DOE  can  help  to  statistically  determine  sample  size  and  the  proper 
number  of  iterations  to  generate  test  results  at  a  predetermined  acceptable  error  and 
confidence  level. 

5.  Researching  which  tools  and  structure  OTC  needs  to  conduct  valid  and  accurate 
testing. 

6.  booking  at  a  recommended  implementation  plan  to  help  OTC  adopt  and  use  DOE 
principles. 

The  final  report  will  expand  upon  all  these  principles  and  use  sample  fictional  databases  to 
derive  statistical  inferences. 


Participants 

J.  Wesley  Barnes,  Cullen  Trust  for  Higher  Education  Endowed  Professor  in 
Engineering,  The  University  of  Texas  at  Austin 

Dr.  Barnes  joined  The  University  of  Texas  at  Austin  in  1974.  A  past  coordinator  of  the 
Graduate  Program  in  Operations  Research  and  Industrial  Engineering,  Professor  Barnes  is 
currently  graduate  advisor  and  chair  of  the  Graduate  Studies  Committee.  Under  Dr.  Barnes’s 
guidance,  the  Operations  Research  and  Industrial  Engineering  Graduate  Program  has  grown  to 
about  80  full-time  graduate  students. 

The  author  of  several  books,  including  a  winner  of  the  Institute  of  Industrial  Engineers  Book- 
of-the-Year  Award,  Dr.  Barnes  has  recently  become  one  of  the  world’s  leaders  in  applying  direct 
search  optimization  techniques  to  problems  of  production  scheduling,  manufacturing  processes, 
vehicle  routing,  and  military  logistics.  Dr.  Barnes  is  the  author  of  over  100  technical  articles  and 
reports  and  has  supervised  20  PhD  students  and  40  MS  students.  In  the  last  sixyears.  Dr.  Barnes 
has  obtained  over  $1,000,000  in  sole-principal-investigator  research  grants  to  support  his 
investigations  into  metaheuristic  approaches  to  complex  logistics,  routing,  and  service  systems 
problems. 

Dr.  Barnes  is  currently  director  of  a  research  consortium  joining  The  University  of  Texas  at 
Austin,  the  Air  Force  Institute  of  Technology,  and  the  Air  Force  Air  Mobility  Command.  He  was 
recently  appointed  adjunct  professor  with  the  Department  of  Operational  Sciences  at  the  Air 
Force  Institute  of  Technology.  The  research  consortium,  in  its  fourth  year  of  activities,  is  funded 


lAT.R  0477 


10 


by  the  Air  Force  Office  of  Sponsored  Research  to  perform  “advanced  air  mobility  command 
operational  airlift  analyses  using  group  theoretic  metaheuristics.” 

Major  John  N.  Whilden,  Graduate  Student,  The  University  of  Texas  at  Austin 

MAJ.  Whilden  hails  from  Grand  Prairie,  Texas.  He  was  commissioned  a  Second  Lieutenant 
in  December  1990  and  has  served  in  the  Army  in  the  United  States,  Korea,  and  Kuwait.  His 
education  includes  a  Bachelor  of  Business  Administration  with  a  specialty  in  management 
information  systems  from  Texas  Tech  University,  a  Master  of  Science  in  logistics  management 
from  The  Florida  Institute  of  Technology,  and  he  will  complete  a  Master  of  Science  in  operations 
research  and  industrial  engineering  from  The  University  of  Texas  at  Austin  in  May  2007.  He  is  a 
graduate  of  the  Chemical  Officer  Basic  Course,  The  Combined  Logistics  Officer  Advanced 
Course,  The  Combined  Arms  and  Staff  Services  School,  and  The  Petroleum  Officers  Course. 

His  awards  and  decorations  include  the  Meritorious  Unit  Citation,  two  awards  of  the 
Meritorious  Service  Medal,  two  awards  of  the  Army  Commendation  Medal,  the  Army 
Achievement  Medal,  the  Army  Reserve  Component  Achievement  Medal,  two  awards  of  the 
National  Defense  Medal,  the  Global  War  on  Terrorism  Expeditionary  Medal,  the  Global  War  on 
Terrorism  Service  Medal,  the  Korean  Defense  Service  Medal,  the  Army  Service  Ribbon,  and  the 
Overseas  Service  Ribbon. 

J.  Toufic  Chahin,  Research  Engineer,  The  University  of  Texas  at  Austin 

Mr.  Chahin  is  a  research  engineer  for  the  University  XXI  team.  He  recently  joined  the  team 
after  filling  positions  as  research  director  at  WIRED  magazine  and  analyst  at  Stanford  Research 
Institute  Consulting  and  Business  Intelligence.  Mr.  Chahin  earned  an  MS  in  science  and 
technology  commercialization  while  attending  the  IC2  Institute  in  2002,  and  a  BA  in 
mathematics  in  2000,  both  from  The  University  of  Texas  at  Austin.  He  is  the  project  lead  for 
Design  of  Experiments,  which  is  a  University  XXI  project  in  support  of  the  Army’s  Operational 
Test  Command  (OTC). 


lAT.R  0477 


11 


